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Outline 


• Motivation 

• Background: (static) SAVE 

• Dynamic SAVE Vision 

• Dynamic SAVE examples 

• Applicability Throughout the Life Cycle 
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Problem/Approach 


• Systems are often difficult to understand 

- Systems of systems adds to the challenge 

- Makes system verification difficult 

- Interfaces often source of problems 

• Approach 

- Architecture analysis focusing on interfaces 

• The new tool, Dynamic SAVE, 

- extends the already existing static Software Architecture 
Visualization and Evaluation (SAVE) tool 
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Background: The (static) SAVE Tool 


Software Architecture Visualization and Evaluation 


• Does the actual implementation match the planned architecture? 

- Define a planned architecture 

- Create an actual architecture from source code 

- Identify architectural violations through comparison 

• Applied to APL’s Common Ground System 

• NASA Research Infusion project (Aerospace 2007) 

- (and other systems, e.g Core Flight System (cfs/cfe,) SNAS, White Sands) 

• Conclusion 

- The SAVE approach is useful and practical 

- One can quickly model, visualize, analyze, find static architecture violations 

- Good for single software applications 

- But for systems of systems, some questions remain unanswered... 
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Dyn-SAVE Vision 
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Specify Level of Abstraction 
For analysis 


Who does socket communicate with? 

Is communication according to specification? 
Check Sequences, Parameters, Values, Timing 
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The Current Work 
On Dynamic SAVE 
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DynSAVE in perspective 
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Focus on: 


Interface Control Documents 


- NASA systems often developed by different teams 

- Interface Control Documents (ICD) is key, but 

• ICDs often interpreted differently because 

• ICDs implicit, lack important details etc. 

- Cause subtle critical deviations from specified behavior 

• Deviations difficult to detect 

• Emerging behavior difficult to predict 

- Can result in severe problems, e.g. lost data, performance 

- Need to 

• Detect deviations before deployment 

• (Specify expected and actual behavior before creating ICD!) 
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Research Questions 


• Sequence diagrams 

- Can we use sequence diagrams to model, 
abstract, and detect such deviations? 

- Can sequence diagrams express what we 
need? 

• Iterative modeling 

- Can we start with abstract models, add details 
as necessary? 
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Approach 


• Collect concrete examples from APL 

- Model planned behavior 

• Use specification from ICD 

- Capture actual traces 

• Use Archive_Server and Eng_Dump 

• Generate Client scenarios, observe how Server 
responds 

• Identify common patterns 
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The “simplest” diagram that describes the planned 
communication behavior described in the ICD 
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Example 1 : Illegal filter 


client 


server 





Fitter 



An illegal extra filter is sent after BeginPlayback and Data messages have been sent. 
The illegal filter is difficult to detect because it is in packet 869. 
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Example 2: Illegal Type specification 
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Adding Timing 


Constraints 
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Checking for Timing Problems 
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CFDP - A Mission Data 


System Protocol 


• CFDP software provides reliable downloads 
of recorded on-board data 

- The implementation is distributed across flight 
and ground systems 

- The protocol runs on top of unreliable CCSDS 
command and telemetry layer 

• At APL, CFDP is mostly automated, but... 

- Operators turn off CFDP uplink during critical 
command load sequences 

- Operators freeze and thaw timers so that 
pending transactions don’t time out between 
contacts 

• Improper CFDP operation can lead to 
unnecessary retransmissions, wasting 
precious downlink bandwidth 
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DynSAVE monitoring of CFDP 


DynSAVE monitors macro-level behaviors of the 
CFDP protocol without affecting flight or ground 
software 


• DynSAVE could detect behaviors that are 
indicative of improper CFDP operation, for 
example: 

- tinners were not frozen and uplink was disabled on 
the ground for an extended period, causing multiple 
retransmissions when the uplink was finally 
enabled again 

• DynSAVE could detect behaviors that are 
indicative of issues in CFDP implementation, for 
example: 

- sender continues to send file data after the 
transaction has been cancelled 

• These types of behaviors can go undetected (file 
transfers still work) but are important to detect 

04-i(£be96can result in data loss!) dynSAVE 


Receiver 



ACK(EOF) 

NAKiM.FDtfj) 


FIN 


ACK(FIN) 

(close) 


{dose) 


23 



Fraunhofer USA, Inc 


Center fw Experimentel 
Software Engineer; nti 
P^arytand 


Planned CFDP Sequence 
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Rules: 

1 .Check that received FD are not NAKed * 

2. Check for duplicate FDs * 

3. Check that we have all FDs upon FIN * 

4. Check that identical NAKs are not sent back-to-back unless tinner went off 
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'Attual CFPD Sequence 
Annotated, Collapsed 


Bender 
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Needed FDs: 502 
Send FDs: 840 

Potential Waste: ~70%? - Further analysis needed. 
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Zoom in on CFDP sequence 


Sender 


Receiver 




Fraunhofer USA, Inc 


Center for Experimental 
Software Engineering 
MarylaflQ 


Life Cycle 


Support 


Initial use of Dyn SAVE 


System 

Architecture 


Use DynSAVE to 
Specify and Test 
Communication 
Add to ICD 


Sub-System 

Development 


Use DynSAVE to 
Develop and Test 
based on ICD 


System 

Integration and Test 


Use DynSAVE to 
test based 
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Create System Architecture 


No Server, No Client Exist 
Use DynSAVE to 

• Specify Planned communication Communication 

- Sequences 

- Parameters, Values 

- Timing constrains 

• Create Tests 

- Correct, Incorrect behavior 

• Specific incorrectness 

• Automatically generate defects 

• Ensure that communication 
protocol can handle all tests 

• Add Diagram, Specification, 

Tests to ICD 

• “Generate” information for ICD 
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Sub-System 

Development 


No Client (or Server) Exist 
Server is built to ICD 
Use DynSAVE to 

• Import Planned spec from ICD 

• Use Tests from ICD, create new 

- Correct and Incorrect behavior 

• Ensure that Server can handle all tests 

• Future research: Generate Mockup 
Clients (exe) for test 

- Remotely controlled Mockup 

• Turn on/off certain Mockup behavior 

- Run simultaneously on several machines 
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Status 


• Dyn-SAVE works for telemetry protocol 

• Currently adding functionality to evaluate 
CFDP protocols 

• Applying Dyn-SAVE to APL’s systems 

• We’d like to apply to other systems 
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Summary 

Analyze, Visualize, and Evaluate 

- structure and behavior using 

- static and dynamic information 

- individual systems as well as systems of systems 

Next steps: 

- Refine software tool support 

- Use approach to review, improve ICD 

• E.g. add planned sequence diagrams, rules to ICD 

- Apply to other systems to get feedback, 
understand needs 



Fraunhofer USA, Inc 


Center fw Experimefrtai 
>3ftrfare En^ineen m 



Architecture Analysis of Evolving 
Complex Systems of Systems 

Executive Status Report 
Software Assurance Symposium 


Principal Investigator (PI): Dr. Mikael Lindvall, FC-MD 
NASA POC: Sally Godfrey, GSFC 
Team members: 

Chris Ackermann, Dr. Arnab Ray, Lyly Yonkwa, Dharma Ganesan (FC-MD) 

William C. Stratton, Deane E. Sibol (APL) 

Fraunhofer Center for Experimental Software Engineering Maryland (FC-MD) 

Fraunhofer Institute for Experimental Software Engineering (IESE) 

Johns Hopkins University Applied Physics Laboratory Space Department Ground Applications Group (APL) 

SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall 


2008 



Fraunhofer USA, Inc 


Center fw Experimefrtal 
>3ftrfare En^ineen m 


Problem/Approach 



• Systems are often difficult to understand 

- Systems of systems adds to the challenge 

- Makes system verification difficult 

- Interfaces often source of problems 

• Approach 

- Architecture analysis focusing on interfaces 

• The new tool, Dynamic SAVE, 

- extends the already existing static Software Architecture 
Visualization and Evaluation (SAVE) tool 
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Capture Dynamic 
Information 


Specify Level of Abstraction 
For analysis 


Who does socket communicate with? 

Is communication according to specification? 
Check Sequences, Parameters, Values, Timing 
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Relevance to NASA 



- NASA systems often developed by different teams 

- Interface Control Documents (ICD) is key, but 

• ICDs often interpreted differently because 

• ICDs implicit, lack important details etc. 

- Cause subtle critical deviations from specified behavior 

• Deviations difficult to detect 

• Emerging behavior difficult to predict 

- Can result in severe problems, e.g. lost data, performance 

- Need to 

• Detect deviations before deployment 

• (Specify expected and actual behavior before creating ICD!) 
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(Interface Control Documents) 
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Current capabilities 



• Applied to APL’s Telemetry protocol 

- See example below 

• Currently Capabilities allows us to 

- Model planned behavior (based on ICD) 

• Sequences, Parameters, Values, Timing 

- Capture and parse actual communication 

- Visualize actual behavior 

- Compare planned behavior to actual 

- Automatically detect and visualize deviations 

• Already detected some surprising deviations! 
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Abstract planned diagram 
for Telemetry protocol 
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The “simplest” diagram that described the planned 
communication behavior described in the ICD. 

Enhance in iterative fashion. 
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Detailed planned & actual 
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STF ordered - 
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More examples and details in technical presentation! 
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Planned capabilities 



Being able to 

• Model Planned behavior of 

- Ground system software 

- Flight software 

- Communication between Ground and Flight 

• e.g. CFDP 

• Visualize actual behavior 

• Compare planned and Actual behavior 

• Automatically detect and visualize deviations 
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Technical challenges 



• Difficult to use existing case tools to create 
planned sequence diagrams, e.g. 

- Most only support basic diagrams 

- Export formats often are not correct, usable 

• Overcoming the problem 

- Provide importers for case tool 

- Provide our own sequence diagram editors 
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Summary 



• Analyze, Visualize, and Evaluate 

- structure and behavior using 

- static and dynamic information 

- individual systems as well as systems of systems 

• Next steps: 

- Refine software tool support 

- Apply to other systems 
-Apply earlier in system life cycle 
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